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Abstract 

We consider /3-Plancherel measures jo] on subsets of partitions - and their 
asymptotics. These subsets are the Young diagrams contained in a (fc,£)-hook, 
and we calculate the asymptotics of the expected shape of these diagrams, relative 
to such measures. We also calculate the asymptotics of the distribution function 
of the lengths of the rows and the columns for these diagrams. This might be 
considered as the restriction to the (k, £)-hook of the fundamental work of Baik, 
Dcift and Johansson |3] . The above asymptotics are given here by ratios of certain 
Selberg-type multi-integrals. 



1 Introduction 

This paper studies the asymptotics of certain " /3-Plancherel" measures on subsets of 
partitions. Let Y n = {A | A h n} denote the partitions of n, and let f x denote the 
number of standard Young tableaux of shape A = (Ai, A2, • • •)• For general references 
regarding partitions, Young diagrams and Young tableaux - see ^S]> p0j - Let < 
j3 G R. Baik and Rains [5] consider the following " /3—PlanchereF measure Mn on Y n : 

Indeed, M% is the so called Plancherel measure on Y n . We generalize to subsets 
r n C Y n , considering the subset of the partitions (i.e. diagrams) in the (k,£) hook: 
Let k, £ > be integers and let T n = H(k,£; n) denote the following subset of Y n : 

H(k,£;n) = {A h n \ \ k+l < £}. 

These subsets arise in the representation theory of Lie groups, algebras and superalge- 
bras, see for example ;6 . The measures p^' k '^ below are the (k, £)— hook restrictions 
of the above measures M.S. 
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Definition 1.1 Let A £ H(k,£;n) and (3 > 0, then 

p <ftM) (A) = p (/W) (A) := (/Y rw - 

Z^fj,eH(k,£;n)\J ) 



1.1 Expected shape 

Given r n C Y n , n = 1, 2, . . . and the probability measures p = {p n }^Li on the T n -s, 
one studies the asymptotics of the expected value (i.e. average length) of the first 
row Ai 3 denoted Xi t E, and similarly for the second row \2,e, etc. Similarly for the 
columns. Explicitly, when T n = H(k,£; n), expected values are given by the following 
definition. 

Definition 1.2 If A h n, we write A = (Ai. n , A2 jra , . . .). Also, A' is the conjugate 
partition of A. Let 1 < p < k and 1 < q < i. The expected value of the p-th row is 
E{\ P ) = X ( pP"\n), where 

A 0W) (n) = E^ H{kAn) \,n^r ^ xfA%) = (A ^, )(n)> A (^ )(n); 

l^\£H(k,l;n)\J > 

Similarly for the expected q-th column 

,l{P;k,t)( \ ^2\eH(k,£;n)K,n' \'{P;k,t), \ , J {f3;k,£) , s \ \ 

\,B ( n ) = T7X\0 flnd A S ( n ) = ( A l£ ( n )> A 2£ (")»•••) 

lu\£H(k,l;n)\J ) 

Of course, one can replace -ff (A;, n) in the above definition by other subsets T n C 1^. 
The case T n — and /3 — 2 (Plancherel) has a long history. Let 

Xahn A l,n ' (/ A ) 2 _ XVn ^l,n - (/ A ) 2 



X\2 



nl 



be the expected value of the first row - for the Plancherel measure M%. Hammersley 
jlUj showed that the limit c = linin-^oo w{n)/^Jn exists. Vershik and Kerov |24j 
proved that c = 2 (independently, Logan and Shepp proved that c < 2). Vershik 
and Kerov - and Logan and Shepp also determined the asymptotics of the expected 
shape A in this case. 

Recently, in a major breakthrough paper, Baik, Deift and Johansson [3] determined 
the distribution function of the asymptotics of the first row, relating it to the Tracy- 
Widom distribution [22], see also (Hj and [2H]- The distribution function for the second 
row is given, by these same authors, in The distribution functions for the general 
rows are given in JTj and see also [Ij for the analogue results for colored 
permutations. The above results also establish deep connections with the theory of 
random matrices JH] • For detailed reviews of these results - see and |2j . 
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The main objective of the present paper is to compute the asymptotics of the above 
expected values (i.e. shapes) A^' (n), as well as the corresponding distribution 
functions. The first term approximation is relatively simple, as we show that for each 
1 < p < k and 1 < q < £ 

see Theorem 14.11 Second term approximations of A^g (n) are introduced and stud- 
ied in Sections El El and they have different values for different rows and for different 
columns. These second-term-approximations are given as ratios of certain Selberg- 
type integrals, see Theorems 15.31 and 16.31 below. 

1.2 Distribution functions 

In Section [7| we introduce and study the asymptotics of the Distribution functions 
Ap k ' e \n, z), Ag ^(n, z) for the lengths of the rows and the columns in H(k,£;n) 
- with respect to the above measures. We are able to calculate, asymptotically, a 
first-term approximation of these functions, but only a conjecture is given, about the 
second term approximations. That first-term approximation is 



where r(k,e),p(z) is given by Equation ((Til) , see Theorem 17.31 
1.3 Comparison with maximal shape 

Given a subset of partitions T n C Y n , one looks for A £ T n with maximal degree /\ 
Call it maximal shape (with respect to T n ) and denote it by X m ax- 

In Sections El El 

and^Jthe expected shapes for /3 = 1, 2 are compared with the maximal shape. When 
r n = Y n , the asymptotics of \ ma x was calculated by Vershik and Kerov [21], and 
by Logan and Shepp ,12 . In particular, they proved that asymptotically, the expected 
shape for (3 = 2 and the maximal shape are the same, and that shape is given by the 
two axes and by the curve 

y = 1 + i x ' — x 2 — arccos x] . (2) 
A comparison with the case T n = H(k, £; n) is intriguing. 

When T n is the fc-strip T n = H(k,0;n), the asymptotics of A with maximal f x was 
calculated in [2], and is given by the curve 

y = \ x ' arcsinx + \/l — x 2 ] . (3) 

When r n = H(k, £; n), the maximal A was given in ^Bl- These results are reviewed in 
Section \8. 21 Consider for example the 'strip' case £ = 0, and denote the maximal A by 
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Knax- Comparing it with A^' ' , the 'Plancherele' expected A in H(k,Q;n), we show 
that these asymptotic shapes are not equal - even in their first raw. Nevertheless, 
numerically Xmax and A^' fc ' ^ are remarkably close, at least in the few special cases 
we check below, see Section 0. 

(k 0) 

Also, the asymptotics of \ m ax for r n = Y n is not the limit case of Xmax as k —* oo, 
but the similarity between @ and © is intriguing. It should be interesting to see 
if the ratios of the Selberg-type integrals, which give the expected shapes and the 
distribution functions for T n = H(k, £; n), are in any way related to the Tracy- Widom 
distributions [22] [22], which give the (Plancherel and the 'involution') distribution 
functions in Y n . 



1.4 RSK 

In the case of pn' k '^ and pn' k '^\ the RSK correspondence provides an interesting in- 
terpretation of the above asymptotics. The RSK (Robinson-Schensted-Knuth) cor- 
respondence a < > (P\,Q\) corresponds a G S n with a pair of standard Young 

tableaux of shape A [2Dj. In the Plancherel case /3 = 2 it relates the above expected 
values of the first row to the statistics of the longest increasing (and decreasing) sub- 
sequences in permutation. For example, when a < — > (P\,Q\), Ai is the length of 
a longest increasing subsequence in a, while X± is the length of a longest decreasing 
subsequence in a. By C. Green's theorem there are similar interpretations for A2, A3, 
etc. For a detailed account of the RSK see [20] • Thus the results in 3 etc. can also 
be stated in terms of longest increasing subsequences in permutations. 

It is well known that a is an involution iff a < — ► (P\> P\)- The analogue Probability 
theory of longest increasing subsequences in involutions in S n is done in [5] . 

Denote by Sk,£- n Q S n the subset of the permutations a G S n such that under the 
RSK correspondence a < — > (P\,Q\), we have A G H(k,£;n). For example, Sfc^n is 
the subset of those permutations in S n where any descending subsequence has length 
< k. Thus, A^'^ is the expected value of the longest increasing subsequence in the 
involutions in Sj.^ n . 



2 Selberg type integrals 

As mentioned above, the main results in this paper involve Selberg-type integrals, 
hence we briefly review these type of multi-integrals. In |19j A. Selberg proved the 
following formula: 



/ ••• / {ui---u n ) x 1 -[(1-ui) ---(l- u n )\ v 1 - I J \Ui -Uj\ 2z dui ■ ■ -du n 

n 

= n 



l<i<j<n 

T(l + kz) ■ T(x + (k- l)z) ■ T(y + (fc - l)z) 
. =1 T(l + z) - T(x + y + (n + k - 2)z) 
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Various integral formulas can be deduced from Selber's integra, see for example |14j 
for the Macdonald-Mehta integras. For example Mehta's integral formula (which was 
a conjecture for some time) 



/ 



-(1/2)(E x?) 



T(l+jz) 



1 '—>■-■■> Yl \xi-Xj\ 2z dx 1 ---dx k = (V2n) k • H Wf+ ~ , 

l<i<j<k j=l ^ ' 



can be deduced from Selberg's formula, see ^3] for details. We call these and re- 
lated integrals "Selberg-type integrals". A connection between the RSK and these 
integrals, as well as with random matrices, appears in \TJ\, [Hj. Since the formulas 
from ^7j) El are needed later, we record it here, together with certain variations of 
these asymptotics and integrals that are also needed below. Let 

= {(xi, • • • , x k ) G M. k I x\ > X2 > ■ ■ ■ > x k and xi H h x k = 0}, 

and more generally, 



n 



{(xi,...,x k ,y 1 ,...,y e )\x 1 >--->x k ; y x > • • • > y t ; ^Xi + ^yj = 0}. 



Theorem 2.1 (Theorem 2.10 in \Tfi) Let j k = (l/V^)^ 1 ■ k k ^ 2 and D k (x) 
ni<i<j< k ( x i ~ x j)> then 



E (/Y 

\eH(k,0;n) 



1 \ (*-l)(fc+2)/4 



where 



I(k,0,P) 



Note that by certain symmetry properties of the above integrand, Q k is transformed 
in [T7j into R k . 

Given A = (Xi jn , \2,n, ■ ■ ■) \~ n, write A Pjn = n/k + c Pt n ■ \fn and denote c Pi „ = c Pt n(X). 
The same arguments in |17j prove the following theorem. 



Theorem 2.2 



E • ^ 

XeH(k,0;n) 



1 \ (fc-l)(fc+2)/4 

•"■ 1 n) 



-I'{k, 0,/3), 



where 



I*(k,0,(3) 



D k (x) ■ e 2 



The (A;,£)-hook analogue of Theorem 12. II is proved in [Hj: 
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Theorem 2.3 (See Theorem 7.18 in [6]) Let 

7M = (l/V^)^- 1 ■ (k + £)( fc2 ^ 2 )/ 2 • (1/2)* 

then 



E (/ A )" 

XeH(k,i;n) 



(fc(k+l)+*(*+l)-2)/4 



where 



I(k,£,(3) 



The same arguments also prove 

Theorem 2.4 Let 1 < p < k, let A = (Ai. n , A2,n, ■ • •) G (A;, n) and define c p , n (\) 



n 



via : A Pj „ = - — jj+Cp. n {\)-\fn. Similarly for 1 < q < £ and c' qn (X) := c 9in (A'). 



Then 



E c ^( A ) • 



1 \ (fc(fc+l)+l(l+l)-2)/4 



where 



L*(k,£,(3) = I x 
Similarly for the sum 



D k (x) ■ D e (y) ■ e-^^yP 



E c U A ) • (/Y, 

Ae-ff(fc,£;n) 



wrai/i £/ie corresponding integral 



I*(k,£,(3) 



k,e 



D k (x) ■ D e (y) ■ e 



Generalizations of Theorem 12.41 - with more general functions of the c Pin (A), c' q n (X) 
- are rather obvious, but will not be given here. 

The same arguments of Section 7 of jH], applied to the asymptotics of both numerator 
and denominator, prove the following theorem - which is needed later. 
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Theorem 2.5 Let z > 0, 



and let 
Then 



H(k,£;n,z) = {A e H(k,£;n) \ X ln , X ln < 



n 



k + e 



{k,i),z = {(xi,.. . ,x k ;yi, ...,ye) e Sl(k,£) I x u Vx < A- 



^\&H{k,e;n,z)(f X )^ 



f a ., \D k (x)-D e (y).e- h i l ^ x ^^Yd(^)(x;y) ' 

We call all the above " Selberg-type integrals", and remark that the above expected 
values and distribution functions are given below as ratio of such integrals. 



3 The main results 

3.1 The expected values 

We study the expected values of the row and of the column lengths in T* n — H(k, £\ n) 
with respect to the measures pi^' k '^ introduced in Definition 11,11 The first term 
asymptotics is given by 

Theorem 3.1 (See Theorem \%~7\ ) Let (3 > andT n = H(k,£;n). For each 1 < p < k, 



A 



(AM) 



(n) 



n 



P> E v "' " k + t 
Similarly, for each 1 < q < I, 



namely lim (A^fp'^(n))/ 



n 



k + £ 



1. 



A 



/(J3;k,e) 
q,E 



(n) ~ 



n 



k + . 



The second term approximations are given as follows. Define c^'^'^{n) and c^^'^ 



via 



A^ M) (n) 



P,E \'"> ~ -j-^J + ^E^O 1 ) " alld C *E A 



lim c 



p,E 



in). 



Similarly for the columns. Then 



Theorem 3.2 (see Theorem \6.S\) Let 1 < p < k, then the limit c^g'^ exists, and is 
given as follows: 



C P ,E 



tt h X P 



D k (xi,...,x k )-e h f( x i+-+ x D 



L D k (x 1 , . . . ,x k ) ■ e~ 



'(^+•••+4) 



(4) 



Similarly for the columns. 
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Equation © (or @) is a consequence of the seemingly more symmetric Equation (jllj) . 

In Sections |H1 19*1 and ITU1 the expected shapes for (3 = 1, 2 are compared with the 
maximal shape \ ma x in few special cases. As mentioned in Section \\.\W this shows a 
different behavior in the hook case T n = H(k, £; n) compared with the general case 

r — y 

1 n — 1 n- 



3.2 The distribution function for the first row 



In Section [7| we study the distribution functions of the length on the rows and the 
columns. We calculate the first term approximations and conjecture the second term 
approximations. Given < z G R, denote 



H(k,£;n,z) = {A G H(k,i;n) | \ ln , X' ln < 



n 



k + 



+ Zy/n}. 



Let 1 < p < k, 1 < q < £■ The distribution of the length of the p-th row as a function 
of z is defined as 



n,z 



E 



XeH(k,£;n,z) \ 



(/ 



\\/3 



and similarly for the columns. Recall fir& A) from Section [2 and denote 



fyfcA* = {fai, • • • ,xk;yi, ■ ■ ■ ,yi) G I xi, yi < 2}. 



Then 



Theorem 3.3 (see Theorem \ 7. °J[) Let (3, z > and denote 



CO 



D k (x) ■ D t (y) ■ e*^ s/f) y ) 



(5) 



Then 



11 



k + e 



In Section 17.21 we make some conjectures about the second term approximations of 
Ap (n, z) and \'^' k '^' '(n, z), both in terms of ratios of Selberg-type integrals. In the 
last three sections (Sections 7, 8 and 9), we calculate some special cases and include 
also some computer calculations. 
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Part I 

Expected shape, the (3— Plancherel 
probability 

4 First term approximation 

Recall the notation ~: Let a n , b n be two sequences of, say, real numbers, and assume 
b n ^ if n is large enough. Then a n ~ b n if rim n ,_). 00 a n /b n = 1. Extend ~ to vectors 
as follows: (a 1)n , . . . , a r , n ) ~ (&i, n , • • • , &r,n) iff ~ &i,n for i = 1, . . . r. 
Following the techniques and arguments in Section 7 of jH], (see also ^7j) we show 

(3'k i) 

below the following first approximation of the expected shape A^: = X E ' . 

Theorem 4.1 Let (3 > and F n = H(k,£;n) and let X^' E {n) and X'^ E k ' £ \n) be 
given by Definition M.tA For each 1 < p < k, 

>$B A ( n ) ~ namd y n 1 ™o( A & M) w)/(-^) =i- 

Similarly, for each 1 < q < £, 



Proof. We sketch the proof for the expected row length in the case I = 0, thus 
showing that 

The main point is that since (3 > 0, both sums in the numerator and the denominator 
of definition 11.21 are dominated by the summands corresponding to the partitions A, 
such that A, = r + c% • y/n and with q-s in a bounded interval. In other words, let 
a > and denote 

n 

H a (k, 0; n) = {A £ 0; n) I Aj = — + q • a/^, where |cJ < a, i = 1, 2, . . . , k}. 

k 

Then 

^2 xeH a (k,Q;n) X P,n ' (/ A ) 
(fc,0;n) 

Writing A P) „ = § + c P) „ • -^/n, the expression in the brackets equals 

(fc,0;n) (f )(/ A ) /3 + J2x&H a (k,0;n) c P,nV"(/Y 
(A;,0;n) a (k,0;n) 

The first summand equals ^ while the absolute value of the second summand is 
bounded by a^fn since all |c p (n)| < a. Since r + ch/n ~ r, it follows that 

which completes the proof in the case ^ = 0. q.e.d. 



,. \W;fe,0)/ \ r ^\eH{k,0;n) X P,n ■ (f ) 

hm Arjj = hm — — _g = hm 



,. ^Aei/ Q (fc,0;") P>'' 

hm 



n^oo 
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5 Second term approximation, the 'strip' case (£ = 0) 

Because of Theorem 14.11 we look for a more subtle approximation of Ap^'^(n), 
namely, we look for the expected deviation - of the form c • y/n - from This 

leads us to introduce the asymptotic expected value below. We begin with the 

'strip' case t = 0. The general (fc,£)-hook case is given in the next section. 

Definition 5.1 Let T n = H(k,0;n) and let 1 < p < k, with X^^'°\n) given by 
Definition \l. 6 A Define c^^'°\n) via the equation 

Thus, when n goes to infinity, 

Remark 5.2 It is not obvious that the limit c^^'°^ = lin^^oo Q^^'°\n) exists. How- 
ever, Theorem \5.tA asserts that in fact, this limit does exist. 

Our aim is to calculate c^^'°\ thus calculating "the second term" in the approxi- 
mation of the expected value of the p-th row- length Definitions O and O 
obviously imply the equation 

(>':/,-. II), , I )l\ H(k,0;n) V» (/ A ) '' II \ 1 

<e (»)= — (6) 



n 



If k = 1, Equation © implies that for any (3 > 0, cf^'^ = 0. 



Theorem 5.3 Let k>2. The limit c^^' ^ = lim n _i. [X3 C^g' (n) exists, and is given 
by 

GB;*,o) _ -fa x p • [ D kfa, ...,x k )- e -f(^+-+4)] d(fc-i)^ 



L> fe (x lj ...,x fc )-e-2(-?+-+^) 



d(*-i)s 



Proof. In Equation (jHJ) write Ai, n = § + Cj >n • -y/n, and consider A-s with Cj bounded 
in some interval: |q| < a for some a > 0. By an argument similar to the proof of 
Theorem 14.11 it follows that as n — > 00, c pE ( n ) * s approximated by the ratio 

l^XeH a (k,0;n) U J 

The proof now follows by applying Theorem 12. II to the denominator, Theorem 12.21 to 
the numerator, then cancelling equal terms. q.e.d. 
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Remark 5.4 Note that the denominator of FJ§ is actually a "Selberg"-or a 
Macdonald-Mehta - integral, which can be evaluated for any (3. For example, let 
(3 = 2. By comparing (F.2.10) with (F 4.5.2) of \T$, deduce that 



D k ( Xl , ...,x k )- e -K^+-+^)j rf(fc-i) 



fc-1 f I \ / 1 



Similarly, by comparing (F.2.10) with (F '.4. 5.1) of Ji7| /, deduce that when = 1, 
D k ( Xl , . . . , x k ) ■ e -^ + - +3 ^] ai k -^x = 



/ ! \*M ! , -vajfe-i 1 * / 1 \ 



For explicit values, recall that T (|) = |, t/iat T(l) = 1, and that T(z + 1) = zT(z). 

6 Second term approximation, the (k, £)— hook case 

We turn now to the general (k, £)-hook case. 

Definition 6.1 Let f3 > and let 1 < p < k and 1 < q < I, with X^^^\n) given by 
Defj,nition \1.2\ Define c^^'^ via the equation 

Similarly for the columns: 

The existence of the limits cf^'^ and c' ^ is asserted by Theorem 16.31 below. 
Definitions 11.21 and 16.11 obviously imply 
Remark 6.2 

(fY n \ 1 

\ L^\eH{k,l;n) \J ) ^ 

and 



n 



Y.\eH(k,e-,n) K,n (f^ 13 _ n \ _L_ 
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Theorem 6.3 Let 1 < p < k, 1 < q < £, then the limits c^^'^ and c^^^ exist, 
and are given as follows. 



C p,E 



L D k (xi,...,x k )-e 



.hM [x 2 + ... +xl) 



and 



-q,E 



Ly q -\D £ (y u ...,y £ ).e-^ + -+y^ 



P 



P 



Thus, as n goes to infinity, 
\,E 



" +ci^ ) -^ and Af/'%)~^ + c^ 



k + £^ p> e 



Proof. We prove for c p ' E - in two steps. 

Step 1. We claim that the limit c^g exists, and is given by 



k + C *> E 



(p-,k,i) 

"p,E 



J^ e)Xp .[D k (x)-D £ (y).e-^^ + ^ 



■/n ( M) D k( x ) ■ D t(v) ■ e 2 



P 



0) 



(10) 



71. 



(11) 



The proof of Equation (|11|) is essentially the same as that of Equation (JJJ) . Its starting 
point is Remark 16.21 ( instead of Equation ©)■ Here we write, for 1 < i < k and for 

l<j<i, 



A, 



n 



+ Cj,n • \fn and X'j 



n 

n = k + e 



+ c 'j,n • V^, 



k + i 

and consider A-s with Cj and c'j bounded in some interval. Now follow a 'hook'- 
generalization of the proof of Theorem 15.31 applying Theorems 12.31 and 12.41 and 
complete the proof of Equation 1)11(1 . 

Step 2. We now transform Ijlljl into ©• Let Ip' k '^ denote the numerator of (|11|) : 



r{P;k,l) 

± p 



D k (x) ■ D t {y) ■ e 



-((*+*)/2)(£ *?+£t/?) 



Setting ^2 Xi = u we have ^ yj = —u, and Ip^ ,k,e ^ = K{u)L(—u) du, where 



K(u) 



M k (x,u) 
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and 

L(-u) = [ \D e (y) • e"(WE " d**" 1 )*. 

JM e (y-u) L J 

Here Mk(x,u) = {(xi, . . . , Xk) \ x\ > ■ ■ ■ > Xk and J2 x i = u } an d similarly, 
Mt(y,-u) = {(2/1,... | yi > • • • > ye and £y 3 - = -«}• 



To evaluate Jp fe , proceed as follows. In K(u) and L{—u) substitute x\ = X{ — 
(u/k), and y'j = yj + (u/i). The Jacobians are = 1, x t = x' t + (u/k); Dk(x') = 
D k (x); Dttf) = D e (y); £x? = + (u 2 /fc) and £yf = £ I/? + («7*)- 

Replacing by Xj and y'j by it follows that 



= J t (x) ■ J 3 (y) ■ A(u) + J 2 {x) ■ J 3 (y) ■ B{u). 



Here 



Jx(x) 



A(u) 



M k (x,0) 



e 2H du, B(u) 



00 U (k + i) 2 u 2 

— ■ e 2fe « era, 

— oo 



D k (x) -e'^^^Y S k ~^x, (Ji(x) = l if fc = l), 



J 2 (x) 



M fc (:r,0) 



and 



J: 



iy) = I 

JM 



Di{x) ■ e 



'dP-Vx, (J 3 (y) = i if 



(M r (x,0) = {(xi, . . . ,x r ) | x\ > ■ ■ ■ > x T and x\ + - ■ - + x r = 0}, etc.). Since, trivially, 
B(u) = 0, deduce that l!f' AC) = J x (x) ■ J 3 (y) ■ A(u). 

Let l(P' k '^ denote the denominator in Equation By exactly the same arguments 
it follows that I^ k ^ = J 2 (x) ■ J 3 (y) ■ A(u). By (HQ) 



_(/3;M) _ Ip _ Jl[x) 



P' E f(ftM) J 2 (x)' 

which is the right-hand-side of Equation ©. This completes the proof. q.e.d. 
Remark 6.4 The denominator-integral in is 



(k,i) 



D k ( Xl ,..., x k ) ■ D e (y u . . . , ju) ■ e-^ + " + *i+yl+-+yt) " d k +e-V(x- y) 



and is calculated explicitly in //6V, sec 7] (here (3 = 2z): 



1 



P 



i 



h{k{k-l)+£{£-l)){p/2)+k+i] 



nti w/2 + 1) • n =i rum + 1) 



r(/?/2 + 1) 

where T is the Gamma function (T(n + 1) = n\). 
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Theorems 15.31 and 16.31 imply 



Corollary 6.5 Let 1 < p < k and 1 < q < i. Then 

c p,'e £) = \jkTi ' c f,E° ] > and similarly cf^ = \fj^j ■ . 

Proof. Let a = and in Equation @ substitute x = y/a ■ v. By routine calcu- 
lations, this substitution transforms the ration of integrals © into the ratio in - 
multiplied by the factor y/a, which completes the proof. 



7 The distribution functions 

7.1 First term approximation 
Definition 7.1 Let z > 0. Denote 

H(k,£;n,z) = {A G H(k,£;n) | Ai, n , A' ljn < ^r^ + V"}- 

Let /3>0, 1 < p < k and 1 < q < £■ The distribution of the length of the p-th row as 
a function of z is defined as 



^\&H(k,l;n,z) ^P,n ' (/ A ) 
SA£-H'(fc^;n)(/ A ) /3 



Af k > e \n,z) = ' . (12) 



Similarly, the distribution of the length of the q-th column as a function of z is defined 
as 

\l{P;k,Z)(„ v \ - ^AgH(fc,f;n, 2 ) \,n ' if ) , , 

\ {n,Z)~ — — j-g . {L6) 

Let A G H(k, I; n, z) with n large, then necessarily Ai, . . . , A^ > I and A^, . . . , X'« > k 
(in fact, Lemma IT2*1 proves a much stronger property) so that Ai, . . . , A^ + A^, . . . , \' e = 
n + kl. Write 



n + kl r n + kl , r 

X p = ~k+T p ' P = ' ' ' ' q= k + l + V V ™, 9 = !' • • • » 1 



and notice that ^ c p + ^ c' g = 0. 

Lemma 7.2 VFii/i £/ie above notations (and n large), 

•2 < c p < z, p = 1, . . . , k and —— —' z — c q — z > q = 1, ■ ■ ■ ,£■ 



k-p+l ' ' t-q + l 
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Proof. Clearly, all c p , c' q < z. Assume for example that 



+ P-1 



i + p-l 



• z, hence also ct, ct_i , . . . , c„ < — 

p+l ' p 



Thus 



£ _|_ r, _ 1 

= ci+- • •+c fc +c' 1 + - • -+4 < -(k-p+1)-- --Z+C1+- • •+c p _i+c' 1 + - • -+4 

fc — p + 1 

< -(^ + p-l)-z + £ + p-l)-z = 0, 
a contradiction. Similarly for c' q . This proves the lemma. 

Recall that 



^(M) 
and 



{(xi, . . . ,x k ,yi, ■ ■ ■ ,ye) \ x\ > ■ ■ ■ > x k ; yx>--->yi\ T, x i + T,yj = °} 



(k,£),z 



{(xi,...,x k ;y 1 ,...,ye) G fi( W ) | »i, yi < z}. For example, ^(2,0), 



{(a?, -a:) | < x < z). 
Theorem 7.3 Let (3, z > and denote 

In 



r {k/)A Z ) 



D k {x) -D e {y) -e-^^^^^Y d^ k+l -^{x]y) 



V. D k (x).D t (y) 



-^(E*?+Ei/?) 



/3 



(14) 



iei 1 < p < k, 1 < q < I and let n — > oo, then 
Af^(n,z), Af^)(n,z). 

In oi/ier words, 



n 



k + i 



r{ k ,i),p{z). 



Jim ^±^. A (^)(n,z) 



lim 



(/ A ) /3 



n^oo n 



n— >oo 71 



(fc/),z 



;n ) 





J n D k (x)-D e (y)-e-^ x2 ^yP & d«+*-V(x; y) 



(15) 



L (k,l) 



Similarly for 



n— >oo ?7, 

Proof is similar to the proof of Theorem 14. II Let X Pin = + c Pi n ■ y/n, then 



k + £ 



n 



■\f*A{n,z) 



\£H{k. 



+ 



n 



Y^\£H(k,t;n)U X ) 13 



By Theorem 12. 5| the first summand approaches r/ k £\p(z) as n goes to infinity, and 
by Lemma 17.21 \cp n \ < b ■ z for an appropriate constant b > 0, therefore the second 
summand obviously goes to zero as n goes to infinity. q.e.d. 

Theorem 17.31 is a first-term approximation of the distribution function. 
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7.2 Conjectures about the second term approximation 

Let 



5 (M),/3 



n,z 



(16) 



As in the proof of Theorem l7..3[ lim n ^oo s^e),/3(n, z) = r^^^(z). Numerical evidence 
suggest the following (vague) conjecture. 

Conjecture 7.4 For all k, I > and (3, z > 0, as n goes to infinity the expression 
[s(k,£)A n > z ) ~ r {k,£)A z )) = ( 17 ) 



n 



k + 



n 



k + 



E 



— (E ^+Et?) 



-(E 



d(fc+<-l)(a;) 



oscillates in some symmetric bounded interval centered at zero. We denote that in- 
terval as (—L(k,£, (3, z), L(k,£, (3, z)), where (we conjecture that) —L(k,£, ft,z) and 
L(k,£, (3, z) are the respective infimum and supremum of the values in Equation 

Definition 7.5 Let (3, z > 0, 1 < p < k and 1 < q < t. Then \p*' k (re, z) is given 
by Equation Similarly for the columns X g k (n, z). Now define cf' k ' e \n,z) 

via the equation 

\f>W(n,z) = ^.r {k/) ^z)+cf^(n,z)^i. 



Similarly for c'q^' k ' e \n, z). We would like to understand the behavior of Cp 3 '^' 1 ' (n, z 
and c'q /3 ' k '^' '(re, z) as n goes to infinity. We consider Cp ' k '^(n, z). 

Note that Definition 17. 11 Theorem 17.31 and Definition 17.51 imply 
Proposition 7.6 



(f3;k,l)i 



(n,z) 



J2xeH(k,£;n) (/ A ) 



n 



k + - 



(18) 



Conjecture 7.7 Recall the interval (—L(k,£,(3,z),L(k,£,f3,z)) from Conjec- 
ture \7.4\ As n goes to infinity, Cp'^ (n, z) oscillates in the interval 
(-L(k,£,(3,z),L(k,£,/3,z)) +s p k/) ' P {z), where 



4 M),/3 (z) = j " (m) - z Xp 



D k (x) ■ D e (y) ■ e-^E^E^jV+^Or) 



/n (W) [Dk(x) ■ D e (y) ■ e 



(19) 
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The proof is based on Conjecture EU as follows. In JTBJ) write A Pi „ = ^ + 
then c^' k ^\n, z) = j4(n) + B(n), where 

\ Z^XeH(k,e-,n) \J ) 
and 

B(n) = . 

By arguments similar to those in previous proofs, lim n _ >00 B(n) = s^^^(z), and by 
Conjecture 17.41 -A(n) oscillates in the interval (—L(k,£, (3, z), L(k,£, (3, z)). 



Part II 

Some special cases 

8 (3 = 2, comparison of expected and maximal shapes 
8.1 Expected shape 

Recall the RSK bijection a < — ► (P\, Q\) and the subsets 

Sk,t;n = W S S n | under the RSK a < — > (P\, Q\), A G H(k,£; n) } 

from Section 11.41 For example if £ = 0, it follows from well known properties of 
the RSK that Sk : o- n are the permutations in S n with longest decreasing subsequence 
having length < k. In general, if er £ S n is of shape A = (Ai,A2,...), then Ai is 
the length of a maximal increasing subsequence in a. Thus, for example, A^p^ (see 
Definition [OJ is the expected length of the longest increasing subsequences in S\.^ n . 

Consider first the case i = 0. Since the case k = 1 is trivial, assume k > 2 (and 
£ = 0). In that case, by Theorem 15. 31 the expected shape in S^ojn is 
A C2;fc,o) ^ + Cl . ^ ; f + C2 .^,...,f + Cfc . y^) , where for 1 < p < k, 



(2;fcj0) _ J nfc x p -[g fc ( a : 1 ,...,x fc )- e -^^+-"+^] 2 d( fc - 1 ) i 



/„. ^(.i,..,x fe )- e -fw + "^) 



(20) 



In general, the expected shape of the permutations in Sk/- n is given by Definition ll.21 
and the asymptotic shape as n —* oo is given by Theorem 16.31 both with (3 = 2. 
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8.2 Maximal f x 

Given a subset of partitions T n C Y n , one looks for A = X max £ L n with maximal 
degree / A , see Section IT31 In the case T n = H(k,0;n), the asymptotics of X max is 
given in which we briefly describe here. The analogue result from ^S], for the 
case Y n = H(k,£; n), is also described below. 

Let Hk(x) denote the k-th Hermit polynomial. It is defined via the equation 

Thus £r (as) = 1, Ht(x) = 2x, H 2 (x) = Ax 2 - 2, H 3 (x) = 4x(2x 2 - 3), H 4 {x) = 
16x 4 - 48x 2 + 12, etc. H k {x) is of de gree k and its roots are real and distinct, denoted 

x\ < x 2 < • • • < x k . 

Also, x± + X2 + • • • + x, = • The following theorem is proved in 

Theorem 8.1 As n — > oo, i/ie maximum max{/ A | A G iT(&, 0;n)} occurs when 

The analogue result for T n = H(k,£; n) is given in |T5] : 
Theorem 8.2 Let A = A^fJ G 

Amf2 = (Ai, . . . , Afc, . . .) and Ami' = (A' l5 . . . , A^, . . .), and assume n — » oo, £/ien 



Theorem 8.2 Let A = A^fJ G if(fc, n) maximize / A in iL n) and write 



and 



n 



.Here < • • ■ < x k are the roots of the k-th Hermit polynomial and similarly for 
the Xj -s. 

8.3 Examples of some (k, 0) cases 

Let T n = H (k, 0; n) and (3 = 2 (< — >rsk S'fe.Ojn) an d compare the expected shape 

(2'k 0) (k 0) 

AV ' with the maximizing shape Xmax- We begin with 
The case k = 2. 

Here the expected shape is given by A^' 2,0 ' ~ + c^^^^/n, | + c^' ' \/rij , where 



18 



(2;2,0) 



f2*2 0*) i ' ' ' 1 1 1 

2 E — —c\ E , and by Equation (f^0|) . cjg = /1/I2 is the ratio of the following 

1 



(2;2,0) 



integrals: 

h 
Thus c 



poo 
-- / X 


~2x ■ e~ 2x2 ~ 


JO 




(2;2,0) _ 

1,E ~ 


-4=, hence 



and 1-2 



roo 


2x ■ e~ 2x2 ' 


lo 





dx 



7T 



A 



(2;2,0) 



- + — . y /K,-- — . y /K\ = (- + 0.5641 Vn , - - 0.56419^) • (21) 
Compare A^' 2 ' ^ with Xmax- Here x^^ = l/\/2, a^ 2,0 ^ = — 1/ a/2, so 



Note: working with Equation @ and with 'Mathematica' we calculated c? E , ( 
For n = 100, 200, 300, 400, 500, 600, 700, 800, 900, the corresponding values of 

(2;2,0)/ v 

c\ E (n) are: 

0.517699, 0.530593, 0.536496, 0.54007, 0.542533, 0.544364, 0.545795, 0.546952 and 
0.547915, agreeing with Theorem 15.31 



The case k = 3 



Here A 



(2;3,0) 



§ + c^iE^^/n > § + c 2"e' V> V™ > f + C 3 Z £ U; V 7 "-) j so we calculate the 



(2;3,0) 



(2;3,0) 



c-s. By Equation ((2*01) . c[ 2 ^' ^ = J1/J2 is the ratio of the following integrals: 



x\ 



n 3 



Z^sOe-i^+^+^l 2 d (2) x and J 2 



n 3 



D 3 (x)e 



By Remark 15.41 with (3 = 2 and /c = 3, J 2 = — ZL ^. We calculate the numerator Ji. 

324V3 

The domain of integration is defined by: x\ > X2 > x$ and X3 = —{x\ + X2), so 
xi > X2 > — (xi + X2). When X2 < 0, that condition is equivalent to X2 > —x\/2. 
When x 2 > 0, that condition is equivalent to X2 < x%. It follows that 



Ji = J J ^ \ X! [(xi - x 2 )(2x 1 + x 2 )(x! + 2x 2 )e-^ a; i +a; 2+-i-2)j j dx2 
After some routine calculations ('Mathematica' was used here) we obtain J\ = r^U - 

288 v 2 

It follows that when k = 3, 



lim Ci 2 p'°^(n) 

n— >oo > 



,(2;3,0) 



9^/3 



8V27T 



0.777362... 



(22) 



By similar calculations it follows that cl 2 ^' ^ = 0, hence " ] 



,(2;3,0) 



9^3 ^ 9^3 

— I ==Vn , — j ;= 

3 Sv 7 ^ 3 3 8^2^ 



,(2;3,0) 

■ 



Thus 
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I + 0.777362 , | , | - 0.777362 
Compare now with Amai- Since H%{x) = Ax(2x 2 — 3), Xg 3,0 ^ = x^ 3,0 ^ = and 

x (3,0) = _^_ Thug 

& ~ ( g + -y=V^ .3.3- -yf V^J = (3 + 0.707107^ .3.3- 0.707107^) 



Note: For n = 200, 300, 400, 500, 700, 850 and 1100, 'Mathemat- 
ica' and Equation @ give the following corresponding values of c^'^'°\n): 
0.719084, 0.729014, 0.735096, 0.739317, 0.74494, 0.747816 and 0.751261, in accor- 
dance with Theorem 15.31 



9 Examples of some (k, £ )— hook cases, (3 = 2 
9.1 The (1,1) case 

By Equation @ c^' 1 ^ = 0. [Alternatively, calculate cf E by Equation (|11[). Here 
^1,1 = {(x,y) I x + y = 0}, so y = —x and —00 <x<oo. Thus cf^ = ^1 (/3) / -^2 (/?) 

where = x e~ 2x2 

follows that (for any /3 > 0) 



where = f^x e 2x2 dx = 0.] Similarly c^ 1 ' 1 "' = by Equation (JTDJ). It 



9.2 The (2, 1) case 
By Corollary 16.51 



Urn cg'V) = eg' 1 ) = ^ • cg'°) = ^ • -L = 0.460659. (23) 
Since c^'^ = —c^g , deduce that 

C ?S' 1] = ~]fl • ^ = -0-460659. (24) 

Since the sum of the coordinates in \^ 2,l \n) is n, it follows that c^ 2 ' 1 ^ = (this 
can also be deduced directly from Theorem 16. 3|) . Therefore 

A (2;2,i) ^ + _ 460659 ^ ) ™ _ 0.460659^, l n/3 
Note: for n = 100, 200, 300, 400, 700 and 1100, Re- 



(2;2,l), 



n): 



mark (|6,2j) and 'Mathematica' give the following values of ' E 
0.45423, 0.45571, 0.456475, 0.456962, 0.457777 and 0.458317, agreeing with 
Theorem 16.31 
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10 Expected shape, (3 = 1 



10.1 The general case 

Since a € S n is an involution iff the RSK yields a < — > (P\, therefore A^' fc '^^(n) 
is the expected shape of the involutions in Sk/ ]n - Note that when T n = Y n (and 
/? = 1) Baik and Rains [5] showed that the expected length of the first row (or of 
the longest increasing subsequence in involutions in S n ) is again 2 v / n, i.e. the same 
as the first row of A max , see [5 (1.5). As we show below, this is not the case when 
T n = H(k,£;n). 

Denote A^'^(n) = X^' k '^\n). We summarize: 

1. Let 1 < p < k, l<q<£, then 

l(k,i) _ ^X£H(k,£;n) A P," " f X , r,(fc/) _ EAeH(fc,^;n) A 9,n ' / A 

2. Define Cpffl via 

X(M) = n + c (M) v^ and 5( fc '^ = lim 

Similarly for and Cq k ^ . Thus, when n — ► oo, 

AM ~ + g( fc '^ • and A ,(M) ~ + c'^ • ^^ 

P> n ^. _|_ ^ ' p V ,l emu. ' v <j,n ^ _|_ £ 9 * 

3. (Theorem 16.31 /3 = 1) Let 1 < p < k, 1 < q < I, then the limits and Cg fc '^ 
exist, and are given as follows. 



; n x? fc (xi, . . . , x k ) ■ c -¥(^+-+*J)d(*-i)(x) 



Similarly for the expected columns: 



^(2/1,- ..,^)-e-^^ + -+^)^-i)(y) 



4. (Corollary E3J /? = 1) We have 



^ = ^ • # .0), and similarly cf ^ = ^ • c'^ ■ 
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10.2 Examples for involutions in Sk,o-,n 

(i.e. p = 1 and t = 0). 

When k = 2, we obtain J °° 2x ■ e~ 2x2 = \ and / °° x2x ■ e~ 2x2 = j^L. 
Thus cf ' 0) = and 



n , V 71 " r- n V 71 " r- 

7= ■ yn , = • \/n 

2 2V2 2 2^ 



I + 0.626657^71 , I - 0.626657^) . (27) 



Note: for n = 200, 400, 600, 800, 1000, 1200 and 1400, Remark IO and 'Mathemat- 
ica' give the following values of cf ' 0) (n) : 0.592086, 0.602049, 0.606506, 0.609175, 
0.611002, 0.612354 and 0.613406, agreeing with Equation (|23|) . 



When k = 3 we have 

By Equation (|23|) . c^' ^ = J1/J2 is the ratio of the following integrals: 

J 1= [ x 1 D 3 (x)e-* ( rt +3 2 +3 ®<0x and J 2 = f D z (x)e~^ x l+ X l+ X l) d^x 
Jn 3 Ju 3 

By Remark 15.41 (3 = 1, k = 3), 3% = We calculate the numerator Ji. Similar to 
the evaluation of Ji in Section f8.3l here 



POO PXl . V 

Ji = y J ^ - i 2 )(2xi + x 2 )(xi + 2x 2 )e- 3 ( :c ? +a; 2+-i^) j dx2 



dsi = 1/18. 



It follows that when k = 3, cf * 0) = c^' 0) = 27^ = 0.846284... Again, * 0) = so 
Cg 3,0 ^ = — c['°\ Thus, when k = 3, 



-ri Th Th \ 

- + 0.846284^/n , - , - - 0.846284^) . (28) 
3 3 3 / 

Note: for n = 200, 400, 600, 800, 1000, and 1200, Remark O and 'Mathematica' 
give the following values of c^'°\n) = c^' ^: 

0.789051, 0.80486, 0.812115, 0.816513, 0.819547 and 0.821802, as predicted by Equa- 
tion (|23)l 
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